High-end film applications require the synthesis of high-quality imagery that is often photorealistic or matches a desired artistic style. One approach for synthesizing high-quality imagery is known as Monte Carlo rendering, which simulates the physical process of how light flows through the scene and exposes a piece of film in the camera. Monte Carlo rendering is advantageous since it is physically correct, i.e., it correctly simulates the physics of how light bounces around the walls and reflects off the surfaces like it does in the real world. Therefore to make a scene look realistic, all that is needed is light sources positioned in the proper places when building the computer model of the scene. Monte Carlo rendering produces a beautiful image that looks realistic because it has correctly simulated how light bounces around in the scene.
Although Monte Carlo rendering can produce beautiful, photorealistic images, a disadvantage is that it usually takes a long time—up to a day per image—to produce high-quality results which make it unacceptable for real-world production environments such as computer-generated film production. Running the Monte Carlo rendering algorithm for just a few minutes by using only a small number of samples per pixel results in an image that is very noisy and unusable for high-end film applications. For example, for a high resolution frame like those required in the movies could take more than a day per frame to calculate the final image. Considering a 2-hour digital movie at 24 frames per second that has 172,800 individual image frames, it would take 473 years to generate all the frames for a movie using Monte Carlo rendering. Movies are iterative creative processes, so shots often have to be done over and over, and since the production schedule is usually around a year, this technology is simply not feasible for feature film production. Therefore, Monte Carlo rendering is too time-consuming and difficult to incorporate into high-end film applications.
Although Monte Carle rendered scenes may look good from any angle, doing something as simple as moving the camera or moving an object on a virtual set would require hours of manual labor to ensure the lighting is correct. Monte Carlo rendering systems can produce beautiful, photorealistic images by simulating light transport through a series of multidimensional integrals at every pixel of the image: integration of the radiance over the aperture of the camera, over the area light sources of the scene, over the time the shutter is open, and even over the pixel for antialiasing. Monte Carlo renderers estimate these integrals by taking many point samples of the scene function—functional representation of the ray-tracing system given a specific scene. This sampling process involves tracing rays with sets of random parameters that correspond to the dimensions of integration, e.g., the moment in time of the ray for motion blur, the position of the ray on the aperture of the camera for depth-of-field, and so on. In path-tracing, the Monte Carlo system integrates over randomly selected paths from the camera's image plane through the scene in order to compute full global illumination effects.
If the scene function is evaluated at enough of these multidimensional samples, the Monte Carlo rendering system will converge to the actual value of the integral, resulting in a physically correct image. Unfortunately, the variance of the estimate of the integral decreases with the number of samples, so a large number of samples are needed to get a noise-free result. Thus, although a very noisy approximation of the final image can be obtained in a few minutes, it takes a long time—as much as a day per image—to get a result that is acceptable for high-end rendering applications. This limits the use of Monte Carlo rendering systems in modern digital film production.
One way to address these problems is to apply a noise reduction filter to the noisy image. However, the fundamental problem is that filters cannot easily determine what is unwanted noise (introduced by the Monte Carlo integration process) and what is valid scene content, since scene content can often have a noisy appearance in and of itself.
Bilateral filters can be applied to preserve edges in the image by blending samples using weights computed from the differences in position as well as sample value, thereby avoiding blending samples together whose values differ greatly. Unfortunately, bilateral filters work poorly for filtering general Monte Carlo noise. As an example, a depth-of-field scene including three quads of noisy textures, with only the closest quad in focus, the Monte Carlo integration process produces noise in the blurred regions because rays from a pixel in these areas hit different parts of the quads and therefore have widely varying colors when rendered at 8 samples/pixel. A bilateral filter that uses the sample color to preserve the texture detail in the in-focus quad would also preserve the depth-of-field noise, because, in this example, the color variations for both are very similar explaining why previous methods that use scene features for bilateral filtering do not work for general Monte Carlo effects. Noise has always been a problem associated with Monte Carlo approaches of image rendering.
Attempts to solve the noise problem associated with Monte Carlo rendering include filtering Monte Carlo noise and reducing the source of Monte Carlo noise. Nonlinear filters have been developed to filter Monte Carlo noise such as alpha-trimmed filters (which discard statistical outliers and average the remaining samples). One proposed nonlinear filter spreads out the contribution of “noisy” samples to smooth out the signal. The noisy samples are identified by finding pixels where the variance is still above a threshold after a certain amount of time. Another proposed attempt to filter Monte Carlo noise includes filtering Monte Carlo renderings by filtering the low-frequency indirect illumination separately from the rest of the image. Filters based on anisotropic diffusion have also been proposed to filter Monte Carlo noise by preserving details in the image using a map of image coherence with color, depth, and normal information.
In addition, a modified bilateral filter has been proposed to compare the range values of a Gaussian-filtered version of the image. Yet filtering global illumination has also been proposed using a geometry-based discontinuity buffer to adjust a filter. More recently, the edge-avoiding À-Trous filter has been proposed that incorporates a wavelet formulation into the bilateral filter including adding additional information such as normal and world position to help identify edges in the scene. Overall, the problem with these approaches to filtering Monte Carlo noise is that scene information such as normals and world positions can be corrupted by Monte Carlo noise in effects such as depth-of-field and motion blur, so filters that rely on variations in these values to preserve scene detail cannot denoise these kinds of scenes. This is why these approaches have all focused on denoising irradiance or other forms of global illumination, where the geometry scene information at each sample is unaffected by the random parameters. What is needed is an approach to reduce Monte Carlo noise that can handle general Monte Carlo effects with the same framework.
The source of the noise in Monte Carlo rendering has also been studied in order to develop algorithms to mitigate the problem. For example, it has been studied how to extend non-uniform sampling patterns from 2D to the number of dimensions of the random parameters in order to improve the quality of the final image. Other attempts to reduce the source of Monte Carlo noise introduced new Monte Carlo-based rendering algorithms with lower variance, such as irradiance caching, photon mapping, and multidimensional light cuts. Other attempts reduced the noise by fitting a smooth basis to the noisy data. Multidimensional sampling and the reconstruction problem has also been studied with respect to the source of Monte Carlo noise. A multidimensional adaptive sampling algorithm has been proposed, which adaptively samples the space in all parameter dimensions and can handle a wide range of Monte Carlo effects, but unfortunately suffers from the curse of dimensionality as the number of parameters grows.
Another general method is known as adaptive wavelet rendering, which positions samples based on the variance of a wavelet basis's scale coefficients and reconstructs the final image using a wavelet approximation. This smooths noisy areas and preserves detail, although it produces wavelet artifacts when the sampling rate is low. Adaptive wavelet rendering claims to distinguish between the two sources of image-space variance—scene features and Monte Carlo noise—using the wavelet coefficients. However, the proposed method would not work for depth-of-field scenes since in image space the Monte Carlo noise is similar to the noisy texture detail.
Other attempts to reduce the source of Monte Carlo noise uses transform domain analysis to optimize the adaptive placement of samples for specific Monte Carlo effects. For example, the Fourier domain has been used to efficiently render depth-of-field effects, while frequency-space analysis has been leveraged to develop a sheared filter and sampling method for motion blur. However, these attempts cannot be applied to various effects. Recently, compressed sensing has been used to reconstruct scene signal assuming that it is sparse in a transform domain. This method is not an adaptive-sampling algorithm, but a post-process reconstruction. Unfortunately, compressed sensing methods still need a considerable number of samples to produce good results. Information theory has also been applied to improve ray tracing using adaptive methods, where the entropy of the color or geometry is used to determine the rendering quality of a part of the image.
What is needed is a rendering approach that overcomes the deficiencies of the prior art. More particularly, what is needed is a rendering approach that removes noise and produces a high quality image in short period of time such as a few minutes. The invention satisfies this demand.